Rewards Analytics and Distribution Dashboard for Quantification Review

This goal of this document is to offer an easy way to process the outputs of the praise reward system and perform an analysis of the resulting token reward distribution. It should be considered a work-in-progress.

Out[6]:

Distribution report for round-11

  • This period covers praise given between 2022-05-01 and 2022-05-14.
  • We allocated a total of 4154 TEC tokens for rewards.
  • Duplicate praise received a weighting of 0.1 the value of the original praise.
  • We assigned 4 quantifiers per praise instance.
  • Praise receiver names were not hidden behind pseudonyms during quantification

Praise Data Visualization

Rating distribution

Since praise gets valued on a scale, we can take a look at how often each value of the scale gets assigned by quantifiers. Note: how to process the duplicate scores? For now, just delete them.

TOP10 highest rating praise

Praise score average: 76.5
FROM Tam2140#9361 TO chuygarcia.eth#6692,reason:
for being the TEC fam who went to Amsterdam and represented our Commons during all those events

Praise score average: 66.75
FROM Tam2140#9361 TO natesuits#4789,reason:
for embracing daddyhood and becoming a father

Praise score average: 66.75
FROM bear100#9085 TO EFRA#7972,reason:
for the work done and the dozens of hours it took to finalize our DAO’s audit

Praise score average: 62.0
FROM Juankbell#7458 TO elessar.eth#7945,reason:
for winning one of the Amsterdam Hackatons for making a CRISPR in SuperFluid that allows you to perform multiple steps in a single transaction, making complex interactions possible with a few lines of code. Truly pioneers, double praise for you guys!

Praise score average: 54.0
FROM liviade#1387 TO Mount Manu#3530,reason:
for transcribing weeks and weeks of Praise from the Community Call: his work is not rewarded enough

Praise score average: 52.33
FROM bear100#9085 TO iviangita#3204,reason:
for the work done and the dozens of hours it took to finalize our DAO’s audit

Praise score average: 52.33
FROM solsista#5370 TO SatoriD#2945,reason:
for creating wonderfully synthesized A/V art from our converstions in 0mega and consciously compressing them into 10 mins - this is magic

Praise score average: 44.0
FROM mattyjee#8621 TO kristofer#1475,reason:
for calling me out when i made a rude comment in a github ticket -- i really appreciate you raising it to me and not just letting it slide

Praise score average: 41.75
FROM Juankbell#7458 TO chuygarcia.eth#6692,reason:
for traveling out of country, or out of continent, sharing the same room, and intermingling work & play during the Amsterdam crypto conferences.

Praise score average: 39.33
FROM kristofer#1475 TO mattyjee#8621,reason:
for placing much effort into investigating the broken duplicate praise ids in praise period 1. Full post mortem at: https://github.com/CommonsBuild/praise/issues/380

Praise Reward Distribution

We can now take a look at the distribution of the received praise rewards. You can toggle the inclusion of the different sources by clicking on the legend.

Praise Giving Distribution

We can also take a look at the distribution of the people giving praise.

Praise Flows

Now for something more fun: let's surface the top "praise flows" from the data. Thanks to @inventandchill for this awesome visualization! On one side we have the top 20 praise givers separately (modifiable by changing the variable n_senders), on the other the top 25 receivers (modifiable by changing the variable n_receivers). The people outside the selection get aggregated into the "REST FROM" and "REST TO" categories.

Out[14]:

Quantifier Data

Let's take a closer look at the quantification process and and see if we can spot any problems:

Praise Outliers

To aid the revision process, we highlight disagreements between quantifiers.

Outliers sort by spreads

Here we generate a table which sorts the praise by the size of the spread between the highest and lowest scores. It gives us an overview of the spread distribution.

For an exhaustive list, take a look at the exported file "praise_outliers.csv" .

This is a visual aid. ATTENTION! If there are several praise instances with similar spread and quant score, all but one end up "hidden" on the chart.

Praise score by quantifier -- outliers among the quantifiers?

Let's see the range of praise scores every quantifier gave to see the behavior difference of quantifiers.

To interpret the box plot:

  • Bottom horizontal line of box plot is minimum value

  • First horizontal line of rectangle shape of box plot is First quartile or 25%

  • Second horizontal line of rectangle shape of box plot is Second quartile or 50% or median.

  • Third horizontal line of rectangle shape of box plot is third quartile or 75%

  • Top horizontal line of rectangle shape of box plot is maximum value.

score displacement: tendency to under/over-scoring?

scoring correlation: how similiar am i scoring with others?

Agreement on duplication

Among 527 praises, 38 (7.20%) do not agree on duplication

Praise instances with disagreements in duplication are collected in 'results/duplication_examination.csv'. To compare, look at the last 4 columns: 'DUPLICATE MSG 1/2/3' and 'ORIGINAL MSG'.

Agreement on dismissal

Among 527 praises, 33 (6.25%) do not agree on dismissal

Praise instances with disagreements in dismissal are collected in'results/dismissal_disaggreed.csv'. You can further look into who dismissed and who did not.